Transition from E+ 23.2.0 to E+ 24.1.0


Foreword

This notebook has two big parts:
  • Part 1: aims to transition all regression tests from one E+ version to the next, and allows you to run each test in both the old and the new version
    • These will take quite some time to run (about 1hr to run the tests in the OLD OpenStudio version, transition the IDFs to the new E+ version and run them, and run the tests in the NEW OpenStudio Version, based on almost 200 files currently)
    • By default it will just copy over the SQL from the regression test to place in OLD_DIR, but if you want to force rerun the IDF in the old E+ version you can.
    • At the end of Part 1, you will have three CSV files, one per version, with the site KBTUs for each test. And you also have an organized tree of VERSION/TEST_NAME/ output directories that have the SQL files we will use for sections 6+.
  • Part 2: aims to analyze the differences between versions
    • Section 3.1 just re-queries all SQL file (or you can reload the three CSV files) to highlight the tests with the biggest site KBTU differences
    • Section 3.2 provides a high-level interface that only requires to pass a test name and it will query the relevant SQL files and produce visualization (tables, grouped bar charts, and heatmaps) to analyze where differences may be coming from
    If you have already run Part 1 successfully once, you only need to run Section 1. and you can jump to Part 2 directly.
 

PART 1: Transition all regression tests and run them in both E+ versions ¶

Part 1 is ommited in the HTML because the interesting part is the output and analysis at the end

PART 2: Analyzing differences ¶

Analyzing E+ runtime ¶

THIS NO LONGER WORKS, last version where it worked was 9.2.0... I don't know where the runtime went in the SQLFile

Concat frames and look at files that failed to run ¶

These are the files were we have some (but not all) failures
Out[14]:
E+ 23.2.0-7636e6b3e9 24.1.0-9d7789a3ac
OS 3.7.0 Transition 3.8.0
python_plugin.osm 1044617.682811 nan nan
python_plugin.rb 1044617.682811 1044617.682811 nan
storage_liion_battery.osm 2089235.365622 nan 2089235.365622
storage_liion_battery.rb 2089235.365622 nan 2089235.365622

Rerun transition files that need expand objects (can't do it in parallel) ¶

First look at where we have deviations ¶

Setting % diff threshold to 0.010%

Deviations in Transition and/or new OpenStudio ¶

No description has been provided for this image

Deviations from Transition to new OS only ¶

Deviations from Transition to new OS only: Differences above 0.010%
Out[27]:
E+ 24.1.0-9d7789a3ac
OS Transition 3.8.0
vrf_fluid_temperature_control.rb 0.004% -0.011%
vrf_fluid_temperature_control.osm 0.004% -0.011%
multi_stage_electric.rb -0.073% 0.165%
multi_stage.rb -0.166% -0.014%
multi_stage_electric.osm -0.270% 0.225%
centralheatpumpsystem.osm -0.557% -0.011%
centralheatpumpsystem.rb -0.777% -0.010%
No description has been provided for this image

Grouped bar chart of differences compared to the old OpenStudio ¶

How to read this chart :

The percentage differences are calculated compared to the Old OpenStudio results for both the transitioned results and the new OpenStudio results.

What you need to pay special attention to is when you don't have the same difference between the Transition to Old OS and the New OS to old OS (meaning the difference is not E+'s fault, but OpenStudio's).

No description has been provided for this image
Out[31]:
OS Transition 3.8.0
centralheatpumpsystem.rb -0.78% -0.79%
centralheatpumpsystem.osm -0.56% -0.57%
heatpump_airtowater_fuelfired.osm 0.37% 0.37%
heatpump_airtowater_fuelfired.rb 0.37% 0.37%
lowtemprad_constflow.osm 0.35% 0.35%
daylighting_no_shades.osm 0.30% 0.30%
daylighting_shades.osm 0.29% 0.29%
photovoltaics.osm 0.29% 0.29%
pv_and_storage_facilityexcess.osm 0.29% 0.29%
transformer.osm 0.29% 0.29%
unitary_systems_airloop_and_zonehvac.osm 0.29% 0.29%
plantloop_avms.osm 0.29% 0.29%
epw_design_conditions.osm 0.29% 0.29%
additional_props.osm 0.29% 0.29%
pv_and_storage_demandleveling.osm 0.29% 0.29%
moisture_settings.osm 0.29% 0.29%
utility_bill02.osm 0.29% 0.29%
utility_bill01.osm 0.28% 0.28%
interior_partitions.osm 0.27% 0.27%
solar_collector_flat_plate_water.osm 0.27% 0.27%
multi_stage_electric.osm -0.27% -0.05%
adiabatic_construction_set.rb 0.27% 0.27%
solar_collector_flat_plate_photovoltaicthermal.osm 0.26% 0.26%
solar_collector_integralcollectorstorage.osm 0.26% 0.26%
water_heaters.osm 0.26% 0.26%
swimmingpool_indoor.osm -0.24% -0.24%
swimmingpool_indoor.rb -0.24% -0.24%
surface_properties_ground_and_solarmult.rb 0.19% 0.19%
surface_properties_ground_and_solarmult.osm 0.19% 0.19%
surface_properties_lwr.rb 0.18% 0.18%
surface_properties_lwr.osm 0.18% 0.18%
multi_stage.rb -0.17% -0.18%
adiabatic_construction_set.osm 0.18% 0.18%
outputcontrol_files.osm 0.18% 0.18%
outputcontrol_timestamp.rb 0.18% 0.18%
outputcontrol_files.rb 0.18% 0.18%
output_objects_2.osm 0.18% 0.18%
outputcontrol_timestamp.osm 0.18% 0.18%
output_objects_2.rb 0.18% 0.18%
photovoltaics_sandia.rb 0.18% 0.18%
photovoltaics_sandia.osm 0.18% 0.18%
output_objects.osm 0.17% 0.17%
output_objects.rb 0.17% 0.17%
window_property_frame_and_divider.osm 0.17% 0.17%
window_property_frame_and_divider.rb 0.17% 0.17%
foundation_kiva_customblocks.osm 0.17% 0.17%
foundation_kiva.osm 0.17% 0.17%
unitary_system.rb 0.16% 0.16%
unitary_system.osm 0.16% 0.16%
pthp_othercoils.osm 0.15% 0.15%
zone_hvac.rb 0.15% 0.15%
zone_hvac_equipment_list.osm 0.15% 0.15%
zone_hvac_equipment_list.rb 0.15% 0.15%
zone_property_user_view_factors_by_surface_name.osm 0.14% 0.14%
unitary_vav_bypass_coiltypes.rb 0.13% 0.14%
unitary_test.osm 0.13% 0.13%
unitary_test.rb 0.13% 0.13%
unitary_vav_bypass_coiltypes.osm 0.13% 0.13%
coil_waterheating_desuperheater.osm 0.13% 0.13%
multi_stage.osm -0.12% -0.11%
coilsystem_dxhx.rb 0.12% 0.12%
coilsystem_dxhx_desiccant_balancedflow.osm 0.12% 0.12%
coilsystem_dxhx_desiccant_balancedflow.rb 0.12% 0.12%
coilsystem_dxhx.osm 0.12% 0.12%
generator_microturbine.osm 0.12% 0.12%
heatexchanger_desiccant_balancedflow.rb 0.12% 0.12%
heatexchanger_desiccant_balancedflow.osm 0.12% 0.12%
roof_vegetation.osm 0.11% 0.11%
roof_vegetation.rb 0.11% 0.11%
exterior_equipment.osm 0.11% 0.11%
exterior_equipment.rb 0.11% 0.11%
space_load_instances.osm 0.10% 0.10%
space_load_instances.rb 0.10% 0.10%
heatpump_varspeed.osm 0.10% 0.10%
heatpump_varspeed.rb 0.10% 0.10%
dsn_oa_w_ideal_loads.rb 0.10% 0.10%
shadingcontrol_singlezone.osm 0.10% 0.10%
shadingcontrol_singlezone.rb 0.10% 0.10%
daylighting_devices.rb 0.10% 0.10%
ExampleModel.rb -0.09% -0.09%
lifecyclecostparameters.rb -0.09% -0.09%
multi_stage_electric.rb -0.07% 0.09%
unitary_vav_bypass_plenum.osm 0.09% 0.09%
baseline_sys02.osm 0.08% 0.08%
chillers_tertiary.osm 0.08% 0.08%
chillers_tertiary.rb 0.08% 0.08%
dsn_oa_w_ideal_loads.osm 0.08% 0.08%
humidity_control.rb 0.08% 0.08%
water_economizer.rb 0.07% 0.07%
unitary_vav_bypass.osm 0.07% 0.07%
unitary_vav_bypass.rb 0.07% 0.07%
zone_control_contaminant_controller.rb 0.07% 0.07%
heatpump_plantloop_eir.osm 0.07% 0.07%
heatpump_plantloop_eir.rb 0.07% 0.07%
zone_control_contaminant_controller.osm 0.07% 0.07%
humidity_control.osm 0.07% 0.07%
surfacecontrol_moveableinsulation.osm 0.07% 0.07%
surfacecontrol_moveableinsulation.rb 0.07% 0.07%
coil_cooling_dx.rb 0.07% 0.07%
coil_cooling_dx.osm 0.07% 0.07%
airloop_and_zonehvac.osm 0.07% 0.07%
generator_windturbine.rb 0.07% 0.07%
generator_windturbine.osm 0.07% 0.07%
sizing_zone_dszad.osm 0.07% 0.07%
sizing_zone_dszad.rb 0.07% 0.07%
shadowcalculation.osm 0.07% 0.07%
shadowcalculation.rb 0.07% 0.07%
coil_waterheating_desuperheater.rb 0.07% 0.07%
daylighting_devices.osm 0.07% 0.07%
coil_userdefined.rb -0.07% -0.07%
coil_userdefined.osm -0.07% -0.07%
zone_hvac2.rb 0.06% 0.07%
chiller_electric_ashrae205.rb 0.07% 0.07%
chiller_electric_ashrae205.osm 0.07% 0.07%
baseline_sys01.osm 0.07% 0.07%
pvwatts.osm 0.07% 0.07%
baseline_sys03.rb 0.07% 0.07%
hot_water.rb 0.07% 0.07%
evaporative_cooling.rb 0.07% 0.07%
ideal_plant.rb 0.07% 0.07%
ideal_plant.osm 0.07% 0.07%
hot_water.osm 0.07% 0.07%
air_chillers.osm 0.06% 0.06%
zone_hvac2.osm 0.06% 0.06%
availability_managers.osm -0.06% -0.06%
baseline_sys06.rb 0.06% 0.06%
fuelcell.rb 0.06% 0.06%
fuelcell.osm 0.06% 0.06%
surface_properties.osm 0.06% 0.06%
pv_and_storage_demandleveling.rb 0.06% 0.06%
baseline_sys08.rb 0.06% 0.06%
utility_bill01.rb 0.06% 0.06%
vrf_watercooled.osm 0.06% 0.06%
daylighting_no_shades.rb 0.06% 0.06%
ems.rb 0.06% 0.06%
epw_design_conditions.rb 0.06% 0.06%
additional_props.rb 0.06% 0.06%
plantloop_avms.rb 0.06% 0.06%
baseline_sys01.rb 0.06% 0.06%
pvwatts.rb 0.06% 0.06%
transformer.rb 0.06% 0.06%
photovoltaics.rb 0.06% 0.06%
utility_bill02.rb 0.06% 0.06%
daylighting_shades.rb 0.06% 0.06%
pv_and_storage_facilityexcess.rb 0.06% 0.06%
solar_collector_flat_plate_photovoltaicthermal.rb 0.06% 0.06%
moisture_settings.rb 0.06% 0.06%
plenums.osm 0.06% 0.06%
plenums.rb 0.06% 0.06%
solar_collector_flat_plate_water.rb 0.05% 0.05%
water_heaters.rb 0.05% 0.05%
airterminal_cooledbeam.osm 0.05% 0.05%
solar_collector_integralcollectorstorage.rb 0.05% 0.05%
availability_managers.rb -0.05% -0.05%
vrf_airloophvac.osm 0.04% 0.04%
vrf_watercooled.rb 0.04% 0.04%
foundation_kiva_customblocks.rb 0.04% 0.04%
vrf_airloophvac.rb 0.04% 0.04%
baseline_sys09.rb 0.04% 0.04%
dist_ht_cl.rb 0.04% 0.04%
setpoint_manager_systemnodereset.osm 0.04% 0.04%
setpoint_manager_systemnodereset.rb 0.04% 0.04%
foundation_kiva.rb 0.04% 0.04%
headered_pumps.rb 0.04% 0.04%
baseline_sys04.rb 0.04% 0.04%
baseline_sys02.rb 0.04% 0.04%
heatexchanger_airtoair_sensibleandlatent.rb 0.04% 0.04%
solar_collector_flat_plate_photovoltaicthermal_bipvt.osm 0.04% 0.04%
solar_collector_flat_plate_photovoltaicthermal_bipvt.rb 0.04% 0.04%
refrigeration_system_2.rb 0.04% 0.04%
refrigeration_system_2.osm 0.04% 0.04%
baseline_sys10.rb 0.04% 0.04%
airloop_avms.osm 0.04% 0.04%
airloop_avms.rb 0.04% 0.04%
ptac_othercoils.osm 0.04% 0.04%
zone_fan_exhaust.rb 0.04% 0.04%
fan_on_off.rb 0.04% 0.04%
refrigeration_system.osm 0.04% 0.04%
refrigeration_system.rb 0.04% 0.04%
fan_componentmodel.rb 0.04% 0.04%
fan_componentmodel.osm 0.04% 0.04%
air_terminals.rb 0.04% 0.04%
schedule_file.rb 0.04% 0.04%
schedule_file.osm 0.03% 0.03%
unitary_vav_bypass_plenum.rb -0.03% -0.03%
surface_properties.rb 0.03% 0.03%
baseline_sys06.osm 0.03% 0.03%
pthp_othercoils.rb 0.03% 0.03%
vrf.rb 0.03% 0.03%
baseline_sys08.osm 0.03% 0.03%
zone_property_user_view_factors_by_surface_name.rb 0.03% 0.03%
baseline_sys05.rb 0.03% 0.03%
plant_op_deltatemp_schemes.rb 0.03% 0.03%
plant_op_deltatemp_schemes.osm 0.03% 0.03%
cooling_coils.osm 0.03% 0.03%
cooling_coils.rb 0.03% 0.03%
baseline_sys04.osm 0.03% 0.03%
baseline_sys07.rb 0.03% 0.03%
thermal_storage.osm 0.03% 0.03%
meters.osm 0.03% 0.03%
meters.rb 0.03% 0.03%
thermal_storage.rb 0.03% 0.03%
plant_op_schemes.osm 0.03% 0.03%
plant_op_schemes.rb 0.03% 0.03%
lowtemprad_constflow.rb -0.03% -0.03%
plant_op_temp_schemes.rb 0.03% 0.03%
plant_op_temp_schemes.osm 0.03% 0.03%
fan_systemmodel.osm 0.03% 0.03%
absorption_chillers.rb 0.03% 0.03%
absorption_chillers.osm 0.03% 0.03%
fan_systemmodel.rb 0.03% 0.03%
unitary_systems_airloop_and_zonehvac.rb 0.03% 0.03%
ideal_loads_w_plenums.rb 0.03% 0.03%
ideal_loads_w_plenums.osm 0.03% 0.03%
zone_hvac.osm -0.03% -0.03%
generator_microturbine.rb 0.03% 0.03%
ducts_and_pipes.osm -0.02% -0.02%
baseline_sys03.osm 0.02% 0.02%
dist_ht_cl.osm 0.02% 0.02%
setpoint_managers.osm 0.02% 0.02%
setpoint_managers.rb 0.02% 0.02%
headered_pumps.osm 0.02% 0.02%
ducts_and_pipes.rb -0.02% -0.02%
tablelookup.rb 0.02% 0.02%
tablelookup.osm 0.02% 0.02%
vrf.osm 0.02% 0.02%
airloop_and_zonehvac.rb 0.02% 0.02%
ptac_othercoils.rb 0.02% 0.02%
coil_waterheating_desuperheater_2.rb 0.02% 0.02%
coil_waterheating_desuperheater_2.osm 0.02% 0.02%
heatpump_hot_water.osm 0.02% 0.02%
heatpump_hot_water.rb 0.02% 0.02%
zone_fan_exhaust.osm 0.02% 0.02%
coolingtowers.rb 0.02% 0.02%
performanceprecisiontradeoffs.osm 0.02% 0.02%
performanceprecisiontradeoffs.rb 0.02% 0.02%
airterminal_cooledbeam.rb 0.02% 0.02%
interior_partitions.rb 0.02% 0.02%
vrf_fluid_temperature_control_hr.rb 0.02% 0.02%
vrf_fluid_temperature_control_hr.osm 0.02% 0.02%
fan_on_off.osm 0.02% 0.02%
fluid_coolers.rb -0.02% -0.02%
fluid_coolers.osm -0.02% -0.02%
dual_duct.osm 0.02% 0.02%
dual_duct.rb 0.02% 0.02%
heatexchanger_airtoair_sensibleandlatent.osm 0.02% 0.02%
baseline_sys09.osm 0.02% 0.02%
baseline_sys10.osm 0.01% 0.01%
coilsystem_waterhx.rb 0.01% 0.01%
coilsystem_waterhx.osm 0.01% 0.01%
hightemprad.osm -0.01% -0.01%
hightemprad.rb -0.01% -0.01%
heatrecovery_chiller.rb -0.01% -0.01%
heatrecovery_chiller.osm -0.01% -0.01%
schedule_ruleset_2012_NonLeapYear.rb 0.01% 0.01%
schedule_ruleset_2012_LeapYear.rb 0.01% 0.01%
schedule_ruleset_2012_LeapYear.osm 0.01% 0.01%
schedule_ruleset_2012_NonLeapYear.osm 0.01% 0.01%
humidity_control_2.rb 0.01% 0.01%
humidity_control_2.osm 0.01% 0.01%
schedule_ruleset_2013.rb 0.01% 0.01%
schedule_fixed_interval.rb 0.01% 0.01%
schedule_fixed_interval_schedulefile.rb 0.01% 0.01%
schedule_ruleset_2013.osm 0.01% 0.01%

Stacked bar chart of differences ¶

How to read this chart :

The percentage differences are calculated compared to the Old OpenStudio results for both the transitioned results and the new OpenStudio results. Then I do

% diff New version = % diff New version - % diff Transition

and plot that as a stacked bar chart. The goal is to more clearly see the differences that are due to OpenStudio by removing the differences due to the new E+.

What you need to pay special attention to is when you see % differences for the new OS.

A cursor along with the total % difference between Old OS and New OS is also plotted . Please see the below example to get a better sense of how the graph is constructed

No description has been provided for this image

Let's do the actual plotting now:

No description has been provided for this image